Comparison between the Smart Triage model and the Emergency Triage Assessment and Treatment guidelines in triaging children presenting to the emergency departments of two public hospitals in Kenya

Several triage systems have been developed, but little is known about their performance in low-resource settings. Evaluating and comparing novel triage systems to existing triage scales provides essential information about their added value, reliability, safety, and effectiveness before adoption. This study included children aged < 15 years who presented to the emergency departments of two public hospitals in Kenya between February and December 2021. We compared the performance of Emergency Triage Assessment and Treatment (ETAT) guidelines and Smart Triage (ST) models (ST model with independent triggers, and recalibrated ST model with independent triggers) in categorizing children into emergency, priority, and non-urgent triage categories. Sankey diagrams were used to visualize the distribution of children into similar or different triage categories by ETAT and ST models. Sensitivity, specificity, negative and positive predictive values for mortality and admission were calculated. 5618 children were enrolled, and the majority (3113, 55.4%) were aged between one and five years of age. Overall admission and mortality rates were 7% and 0.9%, respectively. ETAT classified 513 (9.2%) children into the emergency category compared to 1163 (20.8%) and 1161 (20.7%) by the ST model with independent triggers and recalibrated model with independent triggers, respectively. ETAT categorized 3089 (55.1%) children as non-urgent compared to 2097 (37.4%) and 2617 (46.7%) for the respective ST models. ETAT classified 191/395 (48.4%) admitted patients as emergencies compared to more than half by all the ST models. ETAT and ST models classified 25/49 (51%) and 39/49 (79.6%) deceased children as emergencies. Sensitivity for admission and mortality was 48.4% and 51% for ETAT and 74.9% and 79.6% for the ST models, respectively. Smart Triage shows potential for identifying critically ill children in low-resource settings, particularly when combined with independent triggers and performs comparably to ETAT. Evaluation of Smart Triage in other contexts and comparison to other triage systems is required.


Introduction
Overcrowding is a major global problem in many emergency departments (ED) [1,2].This is compounded by lack of validated triage systems that can help health workers distinguish between low and high-priority patients, along with poor adherence to existing triage protocols [3].Triage is a vital component of effective and efficient emergency care for children presenting to health facilities with varying severity of illness.Triage categorizes patients according to the severity of their illness and designates a level of urgency that includes non-urgent, priority, and emergency cases that require immediate medical attention [4].
Rapid triage of critically ill patients can reduce the waiting time for children needing lifesaving treatment by distinguishing them from non-urgent patients who can safely wait in the queue for assessment and treatment by health workers [5,6].However, in resource-limited settings, triage remains underused, especially in paediatric emergency care, owing to significant barriers such as inadequate staffing, the complexity of guidelines, social and organizational context, and lack of capacity [7][8][9].Consequently, the sickest children are not prioritized, resulting in delayed care and inefficient resource utilization [10].
Several paediatric triage systems have been developed and adopted in high and low-income settings to help frontline health workers improve triage accuracy [11][12][13][14][15].One of the most widely implemented triage systems is the World Health Organization's (WHO) Emergency Triage Assessment and Treatment (ETAT) guidelines, which are recommended for use in resource-constrained settings [16,17].The ETAT system provides a systematic and objective approach to triaging children using clinical signs to identify emergency, priority, and nonurgent cases.However, the implementation of ETAT in clinical practice has faced a myriad of challenges, including a high turnover of trained staff, lengthy and high-intensity training, and limited resources to support implementation [18,19].
A new paediatric triage model, Smart Triage, was recently developed based on the data collected from a hospital in Uganda [20].This is a logistic regression model based on nine variables that were selected from over 100 demographic, vital sign measurements, symptoms, and socio-demographic variables collected upon arrival at the hospital.This model provides the probability of admission.The prediction model addresses some of the challenges associated with the implementation of ETAT by utilizing a limited number of predictors to identify critically ill children and can be used by frontline health workers with limited training and expertise.Moreover, the model can be integrated into digital health platforms to minimize the need for memorization of triage protocols and training of triage staff.To facilitate its use in clinical settings, the triage algorithm incorporates a set of independent emergency and priority triggers based on ETAT triage guidelines [20,21].These triggers include rare emergency conditions and ensure that children who would have been misclassified by the model are safely assigned to either emergency or priority categories [16,20].
It is imperative to evaluate the performance of new triage systems and, if possible, to compare them with existing systems before clinical adoption.Assessment of a triage system's performance should be based on its ability to differentiate between low and high acuity patients as they present to the ED, thus minimizing misclassification of patients.Misclassification of low acuity patients into the high acuity category misuses scarce resources and increases the waiting time for more urgent patients.On the other hand, misclassification of high acuity patients into a low acuity category may result in delayed assessment, diagnosis, and treatment, and potentially poorer outcomes [4,22].Compared with existing paper-based guidelines, electronic triage systems provide valuable information to policymakers and health workers regarding the safety, reliability, and effectiveness of a triage system within a particular setting.Furthermore, potential research gaps and areas of improvement can be identified during the evaluation.
The purpose of this study was to compare the performance of ETAT to Smart Triage in classifying children into triage categories using prospectively collected paediatric data from two Kenyan public hospitals.This comparison can provide validation and performance trade-offs of the new triage model compared with the ETAT algorithm and inform its utility in lowresource settings.

Study design
This was a secondary analysis that utilised baseline data of a multisite clinical study that aimed at developing and exploring the use of a paediatric rapid sepsis trigger (PRST) tool.The study was carried out at the emergency department (ED) of Mbagathi County Hospital and Kiambu County Referral Hospital in Kenya between January 2021 and December 2021.The detailed design and methods of the primary study have been described elsewhere [23].

Study setting
Mbagathi County Hospital and Kiambu County Referral Hospital are first-level referral hospitals located in Nairobi and Kiambu counties, respectively.The outpatient departments in both hospitals serve approximately 20,000 children annually.Each ED is managed by a qualified nurse who triages children and an additional nurse who administers treatment in the emergency room.In addition, one or two clinical officers (equivalent to physician assistants) provide consultation and decide on the appropriate management of the children.Children triaged as emergency cases by hospital staff are transferred directly to the emergency room, while the rest wait in the queue.Both hospitals admit children to a paediatric ward where provision of care is led by a paediatrician and the clinical team consists of medical officers, nurses, and medical and clinical officer interns.Children who are critically ill and require intensive care are referred to a tertiary hospital for specialized care.

Population, eligibility, and study procedures
All children aged < 15 years who presented to the ED with an acute illness were eligible to participate.Children scheduled for immunization, elective surgery, wound dressing changes, or clinical review appointments were excluded.Children presenting to the emergency department with an acute illness on weekdays between 8 am and 5 pm were screened for eligibility by study timekeepers and given a sticker showing their arrival time.A systematic sampling method based on a 30-minute time cut-off was adopted.Research nurses assessed patients' eligibility and determined whether they met the criteria for inclusion in the study.If the first patient in the time cut-off was ineligible, the next patient in the same cut-off was evaluated based on their order of arrival at the hospital.Adherence to the sampling procedure was monitored by a trained nurse who acted as the study coordinator.Written informed consent was obtained from the caregiver or parent of the first eligible patient at each time cutoff, and assent was obtained from children aged > 13 years.Informed consent and data collection for patients in need of emergency care were deferred, and obtained after the child was stable, to avoid delays.
The study nurses performed clinical examinations and collected data using a password-protected custom-built android application installed on a Samsung Galaxy A8 tablet.Heart rate (HR) and blood oxygen saturation (SpO 2 ) were recorded using a Masimo iSpO 2 pulse oximeter connected to the tablet, and respiratory rate was measured using a version of RRate [24] built directly into the Android data collection application.The children were then reviewed by a qualified clinician who independently decided on the appropriate management.The study nurses recorded hospital outcomes including patient disposition from the hospital records and uploaded the data to a secure REDCap database [25] hosted on the KEMRI Wellcome Trust Research Programme (KWTRP) server.Children who were sent home from the ED on the day of enrolment and those who were admitted were followed up via a telephone call seven days after the initial visit or after discharge from the hospital to ascertain the outcome.

Sample size
The sample size for this study was not calculated since it was a secondary analysis of the data collected in the baseline phase of the main study.However, the primary study had a minimum sample size of 833 for this phase [23].

Triage systems
ETAT.This triage system uses clinical signs to assign a triage category depending on the level of illness severity.Frontline health workers identify emergency signs using an "ABCD" method, where A and B symbolize airway and breathing problems, C represents circulation, convulsions, and coma, while D denotes severe dehydration.Children presenting to the hospital with life-threatening problems that require immediate life-saving treatment are assigned to the emergency category, while children requiring urgent review by the health worker (from a set of clinical signs) are assigned a priority category.All other children are classified as nonurgent and can safely wait in the queue (Table 1).
Smart Triage.This logistic regression model incorporates nine predictors (five continuous and four categorical variables) implemented in a digital device [22].The predictors are age, temperature, heart rate, transformed oxygen saturation, mid-upper arm circumference (MUAC), difficulty breathing, pallor, oedema, and parental concern.The selection of variables to be included in the model's development was informed by guidelines established by the Pediatric Sepsis Predictor Standardization (PS2) working group through a rigorous process involving a modified Delphi method and systematic review [26].The model uses low-risk and highrisk thresholds to categorize children into three triage categories (emergency, priority, and non-urgent).The original model has a low-risk threshold of 8% and high-risk threshold of 40%.These thresholds were selected to ensure the model had a high specificity and sensitivity of at least 90%, thus ensuring the detection of high-risk cases and identification of emergency cases, ultimately prioritizing these cases and minimizing delays in providing critical treatment [20].This model was previously recalibrated using the recalibration in the large method using data from Mbagathi County Hospital, resulting in new thresholds of 2.6% and 13% for lowrisk and high-risk thresholds, respectively (manuscript under review).This recalibration method involved re-estimating the intercept of the original model without changing the coefficients of the predictor variables [27].These thresholds were selected to ensure that the model had sensitivity > 80% for identifying high-risk patients.Furthermore, independent emergency and priority triggers (Table 2) were included in a mobile application for the Smart Triage model to allow for its safe clinical implementation.The recalibrated model with independent triggers is currently being implemented and evaluated at the Mbagathi County Hospital in Kenya.Data analysis.Using data collected from children at Mbagathi County Hospital and Kiambu County Referral Hospital, demographic characteristics were summarized as frequencies, percentages, proportions, medians, and corresponding interquartile ranges (IQR).A classification table was used to compare the distribution of participants into three triage categories according to the ETAT guidelines: the ST model with independent triggers, and the recalibrated ST model with independent triggers.The change in the classification of the participants into different triage categories was visualized using Sankey diagrams.Sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) with 95% confidence interval (CI) for admission and mortality were assessed by creating two groups from each triage system: high acuity (emergency) and low acuity (priority and non-urgent).Smart Triage categories were calculated in Microsoft Excel 2016 (Microsoft, Richmond WA) and then data transferred to R 4.3.0(R Foundation for Statistical Computing, Vienna, Austria) for final statistical analysis.
Ethics statement.The study was approved by the Kenya Medical Research Institute (KEMRI) Scientific and Ethics Review Unit (SERU/3958) and Institutional Review Boards at the University of British Columbia in Canada (ID: H19-02398; H20-00484).

Triage assignment
Of the total number of participants enrolled, 513 (9.2%), 1163 (20.8%), and 1161 (20.7%) were identified as emergency cases by the ETAT, ST model with independent triggers, and recalibrated ST model with independent triggers, respectively (Table 4).ETAT categorized the majority of children, 3089 (55.1%), into the non-urgent triage category, unlike the ST model with independent triggers, and recalibrated ST model with independent triggers, which assigned 2097 (37.4%) and 2617 (46.7%) children, respectively, into the non-urgent category.The ST model with independent triggers had a higher proportion of children classified into the priority category, 2373 (42.3%) compared to ETAT and the recalibrated ST model with independent triggers, which only classified 2003 (35.7%) and 1827 (32.6%) children into this category, respectively (Table 4).The ST-only model assigned the highest proportion of children, 2373 (42.3%), to priority category compared to the other triage scales (S1 Table ).The emergency category had the highest proportion of children who were admitted in all triage systems.For children who died, both the ST model with independent triggers and the recalibrated ST model with independent triggers identified 39 (79.6%) as emergency cases, whereas ETAT and the ST-only model identified 25 (51.0%) as such (S1 Table ).

Change in participant classification
Overall.Of the participants classified as non-urgent by ETAT, 20.4%, and 12.6% were reclassified as priority by the ST model with independent triggers, and recalibrated ST model with independent triggers, respectively (Fig 2).Interestingly, 656 (11.7%) of the children who had been categorized as priority by ETAT were classified as emergency by both ST models with independent triggers.ETAT identified 54 (1%) children as non-urgent, but ST models with independent triggers classified them as emergency (Fig 2).Only 7 (0.1%) and 10 (0.2%) children were classified as emergency by ETAT and were reclassified as non-urgent by the ST model with independent triggers, and recalibrated ST model with independent triggers, respectively.A total of 199 (3.6%) and 278 (5%) children who were classified as priority by ETAT and were categorized as non-urgent by the ST model with independent triggers, and recalibrated ST model with independent triggers, respectively.Additionally, 55 (1%) and 52 (0.9%) children who were classified as emergency by ETAT were categorized as priority by the ST model with  Admission.Of ETAT's priority cases, 102/395 (25.8%) of the admitted patients were classified as emergency by both ST models with independent triggers, respectively (Fig 3).A total of 20 (5.1%) and 15 (3.8%) children who were admitted were classified as non-urgent by ETAT were categorized as priority by the ST model with independent triggers, and recalibrated ST model with independent triggers, respectively.Neither the ST model with independent triggers nor the recalibrated ST model with independent triggers categorized any of the ETAT emergency cases as priority.Overall, the majority of admitted patients were classified into the same triage category using the ETAT and ST models (Fig 3).ETAT demonstrated a sensitivity of 48.4% (95% CI: 43.5% -53.3%) for determining hospital admission while both ST models with independent triggers achieved a sensitivity of 74.9% (95% CI: 70.4% -79%) (Table 5).
Mortality.The ST models with independent triggers categorized 34 of the 49 deaths (69.4%) into the same triage categories at ETAT (Fig 4).The addition of independent triggers to the Smart Triage model improved the classification of mortality cases by ensuring that all ETAT emergencies were all still emergencies.Of the 49 patients who died, 13 (26.5%)were categorized as priority by ETAT and emergency by the ST models with independent triggers (Fig ).The change in the classification of ETAT non-urgent patients into other triage categories was similar among ST models with independent triggers (Fig 4).The sensitivity for mortality was 51% (95% CI: 37.5% -64.4%) for ETAT and 79.6% (95% CI: 66.4% -88.5%) for ST models with independent triggers (Table 6).

Discussion
In this study, we compared the ETAT guidelines commonly used in Kenyan public hospitals and the newly developed Smart Triage model based on their ability to classify children presenting to the ED into emergency, priority, and non-urgent categories.The major finding based on the selected model thresholds, was a shift in participants classification from non-urgent to priority to emergency when moving from ETAT to the ST-only model.The magnitude of this shift was larger when independent triggers were added.The emergency category consistently had the highest proportion of children who were admitted or died across all triage systems.The purpose of paediatric triage is to distinguish children who require urgent intervention by the clinical team from those who can safely wait in the queue when they present to the emergency department.As expected in any triage system, the number of emergency cases should be fewer than priority and non-urgent cases.This was the case in our study, where the proportion of children who were classified as emergency was lower than those assigned priority and non-urgent categories by all triage systems.Assignment of fewer patients in the high  acuity category is particularly important to minimize pressure on the already overwhelmed ED.Additionally, the emergency classification expedites clinical review and intervention for children in need of urgent care, thereby improving patient outcomes and ensuring the appropriate use of meager resources.The Smart Triage models caused a shift in the distribution of children across the three triage categories, with more children assigned priority category by the ST model with independent triggers.Additionally, all ST models classified more children into the emergency category compared to ETAT.This was intentionally done with the selection of risk thresholds used to classify patients into these categories, to maximize the models' sensitivity and specificity, and to reduce misclassification during triage [20].Furthermore, as the Smart Triage algorithm is a continuous probability, the proportion of cases in the emergency and priority groups determined by the algorithm can be adjusted to meet the needs of the local context and resources.
Since there is no absolute measure of acuity, admission and mortality have typically been employed in research to assess how effectively a triage system performs [28,29].Similarly, we used these outcomes as proxies for illness severity.An ideal triage system should categorize children in need of admission or at a high risk of mortality in the high-acuity category.Our study findings showed that ETAT and Smart Triage models accurately identified critically ill children, with most admissions and deaths assigned to the emergency category.In addition, incorporation of independent triggers improved the Smart Triage model's ability to classify mortality cases.The ST models with independent triggers assigned all mortality cases to either emergency or priority category compared to ETAT which had some mortality cases assigned to the non-urgent category.
Smart Triage offers a major advantage over ETAT in that it can robustly combine a range of continuous and categorical variables in a prediction algorithm that would not be possible for even the most highly trained clinician.The algorithm can also be tuned to optimize the tradeoffs clinicians perform in clinical practice.In the current implementation, a high specificity was selected to avoid missing critically ill children, especially those in the early stages of critical illness that have not reached the criteria for priority or emergency status using ETAT.This is reflected in the increased number of children in the priority and emergency categories.However, this must be balanced by including too many children in the emergency categories and slowing down access to care for critically ill children with obvious danger signs.Implementing Smart Triage can reduce reliance on memorization, manual processes, and paperwork, common in low-resource healthcare settings.This can lower administrative expenses and allocate resources to other crucial aspects of healthcare delivery.Smart Triage is designed with a simplified approach that utilizes only a few predictors and reduces the need for extensive training.The simultaneous integration of multiple predictors, which on their own would not increase priority, allows for identifying children with early signs of deterioration.Moreover, training resources, such as guides and illustrations, are integrated into the system to facilitate training and provide support during the triage process.
The incorporation of independent emergency and priority triggers into the Smart Triage models improved the categorization of admissions and mortality cases, as evidenced by the higher agreement between the Smart Triage models with independent triggers and the ETAT classification.Most of these triggers are used in ETAT to classify patients into emergency and priority categories and can explain the reason for the improvement in the categorization of admission and mortality cases.Therefore, this suggests that the inclusion of independent triggers as an additional criterion in paediatric triage systems that rely on prediction models is essential to avoid missing critically ill children.The prediction algorithm is not necessary in those children with a single symptom or sign that indicates a critically ill child.A similar argument has been proposed when comparing early warning scores to a single extreme observation [30].

Limitations
This study has some limitations.Enrollment in the study took place on weekdays between 8 am and 5 pm.The distribution of children in different triage categories may differ from that at other times of the day and on weekends.We used proxies for critical illness such as admission due to lack of a gold standard for patient acuity which could be influenced by other factors, rather than clinical indications alone.Increase in the number of emergency cases when using Smart Triage may suggest over-triaging by the triage system.However, the number of emergency and priority cases can be adjusted according to the hospital's capacity, as the Smart Triage algorithm uses different thresholds to classify patients into triage categories.Data on the variable "obstructed/absent breathing" was not collected during the baseline phase of the primary study, hence not available for this analysis.Even though this is a rare presentation, it may have led to fewer patients being classified as emergency by ETAT.Lastly, the study focused on evaluating the triage systems' ability to classify children into triage categories and did not assess other important factors, such as resource utilization.

Conclusion
Smart Triage compares well with ETAT and has potential as an efficient system for triaging children and identifying children in need of urgent care, especially when integrated with independent triggers.The Smart Triage algorithm increased the number of children in the priority and emergency groups, but marginally reduced the number of non-urgent children who were admitted or died.The addition of independent triggers to the Smart Triage model further improved the classification of children at high risk of admission and death.Further research is recommended to confirm these findings in other settings, such as primary care, and to compare Smart Triage with other triage systems as well as evaluate its impact on resource utilization.

Table 1 . Clinical signs and categories of the ETAT triage system and how they were mapped to our dataset. Presenting clinical signs used in ETAT triage Presenting clinical signs available in the study data Intervention
(age < 2 months = RR > 60 breaths per minute age 2-11 months = RR > 50 breaths per minute age � 12 months = RR > 40 breaths per minute) A child without any of the above signs https://doi.org/10.1371/journal.pdig.0000408.t001
a Percentage is based on the total number of participants enrolled.bPercentage is based on the total number of admissions.https://doi.org/10.1371/journal.pdig.0000408.t003